Home Photo Content Modeling for Personalized Event-Based Retrieval

ecause digital cameras are so easy to use, consumers tend to take and accumulate more and more digital photos. Hence they need effective and efficient tools to organize and access photos in a semantically meaningful way without too much manual annotation effort. We define semantically meaningful as the ability to index and search photos based on the purposes and contexts of taking the photos. From a user study 1 and a user survey that we conducted, we confirmed that users prefer to organize and access photos along semantic axes such as the event (for example, a birthday party, swimming pool trip, or park excursion), people (for example, myself, my son, Mary), time (for example, last month, this year, 1995), and place (for example, home, Disneyland, New York). However, users are reluctant to annotate all their photos manually as the process is too tedious and time consuming. As a matter of fact, content-based image retrieval (CBIR) research in the last decade 2 has focused on general CBIR approaches (for example , Corel images). As a consequence, key efforts have concentrated on using low-level features such as color, texture, and shapes to describe and compare image contents. CBIR has yet to bridge the semantic gap between feature-based indexes computed automatically and human query and retrieval preferences. We address this semantic gap by focusing on the notion of event in home photos. In the case of people identification in home photos, we can tap the research results from the face recognition literature. We recognize that general face recognition in still images is a difficult problem when dealing with small faces (20 × 20 pixels or less), varying poses, lighting conditions, and so on. However, in most circumstances, consumers are only interested in a limited number of faces (such as family members, relatives, and friends) in their home photos, so we might achieve a more satisfactory face recognition performance for home photos. With advances in digital cameras, we can easily recover the time stamps of photo creation. Industrial players are looking into the standardization of the file format that contains this infor-mation—for example, the Exchangeable Image File Format version 2.2 Similarly, with the advances in Global Positioning System technology, the camera can provide the location where a photo was taken (for example, the Kodak Digital Science 420 GPS camera). Home photo event taxonomy We define home photos as typical digital photos taken by average …